Gradient Descent

Orthogonal to lines of equal cost.
Look for direction of steepest descent and take a step.
Can use a positive semi-definite matrix to move more carefully as the learning rate for more advanced mechanisms.
Stochastic Gradient Descent